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Branching fraction measurements of charmless B° — > K* h^h2 (/ii,2 = K, it) decays are pre- 
sented, using a data sample of 383 million T(4S) — * BB decays collected with the BABAR detector 
at the PEP-II asymmetric-energy B-meson factory at SLAC. The results are: B(B° — > K*° K + K~) 
= (27.5 ± 1.3 ± 2.2) x 10" 6 , B(B° -> K*°n + K-) = (4.6 ± 1.1 ± 0.8) x 10~ 6 and B(B° 
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K*°-k + -k ) = (54.5 ± 2.9 ± 4.3) x 10 6 . The first errors quoted are statistical and the second are 
systematic. An upper limit is set for B(B° -> K*°K + ir~) < 2.2 x 10" 6 at 90% confidence level. 
We also present measurements of CP- violating asymmetries for the observed decays. 

PACS numbers: 13.25.Hw, 12.15.Hh, 11.30.Er 



Charmless decays of P mesons to three-body final 
states are important probes of the weak interaction and 
the complex quark couplings of the Cabibbo-Kobayashi- 
Maskawa (CKM) matrix Improved experimental 

measurements of these charmless decays, combined with 
theoretical developments, can provide significant con- 
straints on the CKM matrix elements and potentially un- 
cover evidence for physics beyond the Standard Model. 
For example, the branching fraction of the decay P° — > 
K * q tt + K~ is sensitive to the CKM matrix elements V t d 
and V u t (see Fig.[IJ. Additionally, a branching fraction of 
the Standard Model suppressed decay B° — > K*°K + w~ 
comparable or larger than that of P° — > K*°tt + K ~ would 
be an indication of new physics. 



In this paper, branching fractions of B Q — ► K*°K + K 




FIG. 1: (a) Penguin and (b) tree Feynman diagrams for the 
decay B° — + K*°ir + K~ . Similar Standard Model diagrams 
do not exist for the AS" = 2 decay B° -> K*°K + 7T~. 

Neutral P-meson decays to ~tt~ hfh^ (where hi^ = 
K or 7r) are dominated by K*°h^ , but can also pro- 
ceed via a four-body nonresonant component, through 
intermediate charmless resonances such as B° — > K*°<j) 
or or other as-yet-unobserved in- 

termediate charmless resonances. To date, there are only 
limits on the inclusive decays P° — > K* hfh2, measured 
by the ARGUS experiment using less than 0.2 fb _1 [5(, 
whilst there are more accurate measurements of charged 
P-meson decays to K^K^h^ 

We identify K*° mesons through their decay to K + tt~; 
consequently, the states P° — > K^hfh^ are self-tagged 
as the charge of the daughter kaon from the K* reflects 
the flavor of the B° meson. The CP-violating decay rate 
asymmetry is defined as 



A 



K' a h~h 



+ 

1 ""2 



K*°h7h~ 



(1) 



B° 



K* 



K- and B° 



are measured, 



where T is the partial B decay width. 



and an upper limit is set for the Standard Model sup- 
pressed decay B° -> K*°K+tt- Q. The selection crite- 
ria require events with a reconstructed K + n~ K{ final 
state, so that the total charmless contribution (including 
resonant charmless substructure) to the K^h^K^ Dalitz 
plot can be measured. The AK*h 1 h 2 values for the ob- 
served decays P° -► K*°K+I<-, B° -> K* q -k + K~, and 
P° — > K*°ir + ir~ are also measured. 

The data on which this analysis is based were collected 
with the BABAR detector [§] at the PEP-II asymmetric- 
energy e + e~ storage ring. The BABAR detector con- 
sists of a double-sided five-layer silicon tracker, a 40-layer 
drift chamber, a Cherenkov detector, an electromagnetic 
calorimeter, and a magnet with instrumented flux return 
(IFR) consisting of layers of iron interspered with re- 
sistive plate chambers or limited streamer tubes. The 
data sample has an integrated luminosity of 348 fb _1 
collected at the T(4S) resonance, which corresponds to 
(383 ± 4) x 10 6 PP pairs. It is assumed that the T(4S) 
decays equally to neutral and charged P-meson pairs. In 
addition, 37 fb _1 of data collected 40 MeV below the 
T(4S) resonance are used for background studies. 

Candidate P mesons are reconstructed from four 
tracks that are consistent with originating from a com- 
mon decay point within the PEP-II luminous region. 
Each of the four tracks is required to have at least 12 
hits in the drift chamber and a transverse momentum 
greater than 100 MeV/c. The tracks are identified as ei- 
ther pion or kaon candidates, with protons vetoed, using 
Cherenkov-angle information and ionization energy-loss 
rate measurements (dP/dx). The efficiency for kaon se- 
lection is approximately 80%, including geometric accep- 
tance, while the probability of misidentification of pions 
as kaons is below 5% up to a laboratory momentum of 
4 GeV/ c. Muons are rejected using information predom- 
inantly from the IFR. Furthermore, the tracks are re- 
quired to fail an electron selection based on their ratio 
of energy in the calorimeter to momentum in the drift 
chamber, shower shape in the calorimeter, AE/dx, and 
Cherenkov-angle information. 

To characterize signal events, three kinematic vari- 
ables and one event-shape variable are used. The 
first kinematic variable, AP, is the difference between 
the center-of-mass (cm.) energy of the P candidate 
and \/s/2, where y/s is the total cm. energy. The 
second is the beam-energy-substituted mass tties = 
\J (s/2 + p, • ps) 2 /P,f — p^, where ps is the P momen- 
tum and (P^ p^ is the four-momentum of the T(4S) in 
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the laboratory frame. The third kinematic variable is the 
K + ir~ invariant mass, to^.o, used to identify K*° can- 
didates. B Candidates are required to be in the ranges 
\AE\ < 0.1 GeV, 5.2500 < to ES < 5.2889 GeV/c 2 , and 
0.776 < m K ,o < 0.996 GeV/c 2 . The event-shape vari- 
able is a Fisher discriminant T constructed as a lin- 
ear combination of the absolute value of the cosine of 
the angle between the B candidate momentum and the 
beam axis, the absolute value of the cosine of the angle 
between the thrust axis of the decay products of the B 
candidate and the beam axis, and the zeroth and second 
angular moments of energy flow about the thrust axis of 
the reconstructed B. 

Continuum quark production (e + e _ — > qq, where q 
— u,d,s,c) is the dominant source of background. It is 
suppressed using another event-shape variable, |cos#t|, 
which is the absolute value of the cosine of the angle 
0t between the thrust axis of the selected B candi- 
date and the thrust axis of the rest of the event. For 
continuum background, the distribution of |cos#t| is 
strongly peaked towards 1.0 whereas the distribution is 
essentially fiat for signal events. Therefore, the relative 
amount of continuum background is reduced by requiring 
|cos6» T | < 0.8. 

Monte Carlo (MC) events that are at least a 1000 
times the number expected in data are used to study 
background from other -B-meson decays. The largest B- 
background for B° — ► K*°ir + 7r~ candidates comes from 
decays including charmonium mesons, such as J/ipK*°, 
XcoK* and ip(2S)K*°, where charmonium decays to 
are misidentified as decays to 7r + 7r~, or where 
the charmonium decays directly to tt + tt~. These back- 
ground events are removed by vetoing reconstructed 
7r + 7r~ masses in the ranges 3.00 < m 7V + 7V - < 3.20 GeV/c 2 , 
3.35 < m^+ v - < 3.50 GeV/c 2 and 3.60 < m w + n - < 
3.78 GeV/c 2 , corresponding to the J/ip, XcQ and ip(2S) 
meson masses, respectively. For B° — > K*°K + K~ can- 
didates, XcoK* events are removed by rejecting events 
with a reconstructed invariant mass in the range 3.32 < 
m K+K - < 3.53 GeV/c 2 . 

Potential charm contributions from B Q — ► D~(-^ 
K*°li2)hi events are removed from corresponding B° — > 
K*°h^h2 candidates by vetoing events with a recon- 
structed K* ^ invariant mass in the range 1.83 < 
mK*h < 1-91 GeV/c 2 . To remove background from D° 
mesons, a veto is applied to any Ktt pair with an in- 
variant mass in the range 1.83 < toa'tt < 1-91 GeV/c 2 
for each B° — * K* h~ili^ decay. Studies of MC events 
show the largest remaining charmed background is B° — > 
D~(-^ K* tt~)it + , with 11% passing the veto. Surviving 
charmed events have a reconstructed D mass outside the 
veto range as a result of using a wrong 7r~ or K + candi- 
date that is incorrectly selected from the other B decay 
in the event. 

A fraction of events for all decay modes have more 
than one B° candidate reconstructed. For those events, 



the candidate with the smallest x 2 °f the fitted B de- 
cay vertex is selected. Studies of MC events show the 
selection of B° — > K*°ir + K~ events produce the largest 
number of multiple candidates, in 21% of events, where 
for these multiple candidates the correct one is selected 
65% of the time. 

After all requirements have been applied, there are five 
main sources of remaining B background: two-body de- 
cays proceeding via a charmonium meson; two and three- 
body decays proceeding via a D meson; combinatorial 
background from three unrelated particles (K *°/i^/i^~); 
charmless two or four-body B decays with an extra or 
missing particle and three-body decays with one or more 
particles misidentified. Along with selection efficiencies 
obtained from MC simulation, existing branching frac- 
tions for these modes 1(J II | are used to estimate their 
background contributions that are included in fits to 
data. 

In order to extract the signal event yield for the channel 
under study, an unbinned extended maximum likelihood 
fit is used. The likelihood function for N candidates is 



1 / m \ n / m \ 
— exp^-^n.J Yl ^2mPi(3,xi)J 



(2) 



where M = 3 is the number of hypotheses (signal, contin- 
uum background, and _B-background), n% is the number 
of events for each hypothesis determined by maximiz- 
ing the likelihood function, and Pi (a, xj ) is a probabil- 
ity density function (PDF) with the parameters a and 
x = (toeSj A-E, T and m K *a). The PDF is a product 
Pi{a,S) = Pi(d mES ,mEs) x P t (a AE ,AE) x P^a^,^) x 
Piioim t0 i m K* )- Studies of MC simulations show that 
correlations between these variables are small for the sig- 
nal and continuum background hypotheses. However, for 
B-background, correlations are observed between toes 
and AE, which are taken into account by forming a 2- 
dimensional PDF for these variables. 

The parameters for signal and -B-background PDFs are 
determined from MC simulation. All continuum back- 
ground parameters are allowed to vary in the fit, in or- 
der to help reduce systematic effects from this dominant 
event type. Sideband data, defined to be in the region 
0.1 < AE < 0.3 GeV and 5.25 < m ES < 5.29GeV/c 2 , as 
well as data collected 40 MeV below the T(4S) resonance, 
are used to model the continuum background PDFs. For 
the toes PDFs, a Gaussian distribution is used for sig- 
nal, a threshold function [12j for continuum and a two- 
dimensional histogram for B-background. For the AE 
PDFs, a sum of two Gaussian distributions with distinct 
means is used for the signal, a first-order polynomial for 
the continuum background and a two-dimensional his- 
togram is used for B-background. The signal, contin- 
uum and _B-background T PDFs are described using a 
sum of two Gaussian distributions with distinct means 
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and widths. Finally, for m^-.o PDFs, a sum of the Breit- 
Wigner function and a first-order polynomial describes 
the signal, continuum, and B-background distributions. 
The first-order polynomial component of the ttlk *o PDFs 
is used to model misreconstructed events in signal and 
background. Within the m K *o fit range, there is also the 
possibility of i?-background contributions from nonreso- 
nant and higher K*° resonances, which are modeled in 
the fit using the LASS parameterization 1J, |l4j. The 
contribution from this background is estimated by ex- 
trapolating a Kit invariant mass projection fitted in a 
higher-mass region (0.996 < m K ,o < 1.53 GeV/c 2 ) into 
the signal region. This estimated background is modeled 
in the final fit into the signal region, and assumes there 
are no interference effects between the Kir background 
and the if* (892) signal. 

Branching fractions B are usually calculated with the 
equation B = n S i g / (N gB -xe), where n s i g is the fitted num- 
ber of signal events n\, e is the average signal efficiency 
obtained from MC simulation and N BB - is the total num- 
ber of BB events. For the B° — > K* hf}i2 branching 
fraction, the average efficiency cannot be taken directly 
from MC events. This is due to the efficiency variations 
across the Dalitz plane and because the distribution of 
events in the Dalitz plane is a priori unknown. To calcu- 
late the branching fraction, a weight is assigned to each 
event j as 



Wj 



(3) 



J2k n k p k(a, Xj) 

where is the row of the covariance matrix associ- 
ated with the rt\ parameter, obtained from the fit [l5j |. 
This procedure is effectively a background subtraction 
where the weights have the property Wj = n S ig- The 
branching fraction is then calculated as B = • Wj/ (ej x 



N 



where tj (a function of m 



and the 



K*° — > K + tt~ decay helicity angle) varies across phase 
space and is simulated in bins using over eight million 
MC events for each channel. The sizes of the bins are op- 
timized to provide continuous coverage of the efficiency 
distribution. 

Figure [2] shows the fitted ttjes projections for the 
5° -> K*°K+K-, B° -► K*°tt+K-, B° -> K*°K+ir-, 
and B° — > A'*°7r + 7r~ candidates, while the fitted signal 
yields, measured branching fractions, upper limits and 
asymmetries are shown in Table HJ The candidates in 
Fig. [2] are signal-enhanced, with a requirement on the 
probability ratio V a i g /(V S i S + Pbkg), optimized to en- 
hance the visibility of potential signal, where V s i g and 
Pbkg are the signal and the total background probabil- 
ities, respectively (computed without using the variable 
plotted). The 90% confidence level (C.L.) branching frac- 
tion upper limit (<8tjl) is determined by integrating the 
likelihood distribution (with systematic uncertainties in- 
cluded) as a function of the branching fraction from to 



£ UL , so that J™ CAB = 0.9 / °° CdB. The signal signif- 
icance S is defined as \/2A ln£, where A In £ represents 
the change in log-likelihood (with systematic uncertain- 
ties included) between the maximum value and the value 
when the signal yield is set to zero. 



> 
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uu 




1000- 
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FIG. 2: Maximum likelihood fit projections of vties for signal- 
enhanced samples of charmless B° — » K*°hffi2 candidates. 
The dashed line is the fitted background PDF while the solid 
line is the sum of the signal and background PDFs. The points 
indicate the data. The plot shows projections for (a) B° — » 
K* K + K~, (b) B° K*°tv+K~, (c) B° -> K*°K+tv- and 
(d) B° -> K*°-k+-k-. 



Contributions to the branching fraction systematic un- 
certainty are shown in Table [Til Errors due to track- 
ing efficiency are assigned by comparing control chan- 
nels in MC simulation and data. The error in the effi- 
ciency is due to limited MC statistics of the signal event 
samples. The number of BB events is calculated with 
an uncertainty of 1.1%. To calculate errors due to the 
fit procedure, a large number of MC samples are used, 
containing the numbers of signal and continuum events 
measured in data and the estimated number of exclu- 
sive ^-background events. The differences between the 
generated and fitted values are used to estimate small 
fit biases (see Table fll]) that are a consequence of cor- 
relations between fit variables. These biases are ap- 
plied as corrections to obtain the final signal yields, and 
half of the correction is added as a systematic uncer- 
tainty. The uncertainty of the i?-background contribu- 
tion to the fit is estimated by varying the known branch- 
ing fractions within their errors. Each background is var- 
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TABLE I: Signal yields, B-background yields (B bkg), efficiencies, and branching fractions (B), measured using K + tt~ hfh^ 
events. Fit bias corrections are applied to the signal yields and branching fractions. The first error is statistical and the second 
error is systematic. The efficiencies take into account B(K*° — > K + ix~) — 2/3, assuming isospin symmetry. The significance 
S is shown for B° — > K*°tv + K~ and B° — > K* K + tt" and the branching fraction upper limit at 90% C.L. is shown for 
B° — > K*° K + ir~ . Asymmetries (Ax'hh) are reported only for the channels with significant yields. 





Mode 


Total 


B bkg 


Signal 


Signal 


B 


Bvl 


S(a) 


Asymmetry {A K *hh) 






Events 




Yield 


Efficiency(%) 


(x KT 6 ) 


(x 10" 6 ) 






B u 


-> K* U K+K- 


11185 


221 


984 ± 46 


9.3 


27.5 ± 1.3 ± 2.2 




> 10 


0.01 ± 0.05 ± 0.02 


B° 


K*°ir+K- 


53991 


276 


183 ± 42.4 


10.4 


4.6 ± 1.1 ± 0.8 




5.3 


0.22 ± 0.33 ± 0.20 


B° 


K*°K+7T- 


16248 


372 


18.8 ± 29.4 


9.8 


0.5 ± 0.8 ± 0.5 


2.2 


0.9 




B° 


-> #*°7r+7r- 


100750 


1701 


2019 ± 108 


9.7 


54.5 ± 2.9 ± 4.3 




> 10 


0.07 ± 0.04 ± 0.03 



ied individually and the effect on the fitted signal yield 
is added in quadrature as a contribution to the uncer- 
tainty. The higher K*° background is varied by its un- 
certainty and the largest change to the signal yield is 
added as a systematic error. The uncertainty due to 
reconstructing the wrong B° — > K^h^h^ signal can- 
didate as a consequence of K/n misidentification (for ex- 
ample B° — > K*°K + K~ events being reconstructed as 
B° — > K*°n + K~) is determined using MC events and 
added as a systematic uncertainty. We compute uncer- 
tainties and corrections to MC using the high statistics 
calibration channel B° -> D~(-> K m tt-)h + . Over 7000 
events are selected using the B° — ► K* tt + tt~ selection 
criteria and requiring the reconstructed K*°tt~ invariant 
mass to be in the range 1.84 < m K ,o v - < 1.88GeV/c 2 . 
The uncertainty due to PDF modeling is estimated from 
the calibration channel and by varying the PDFs accord- 
ing to the precision of the parameters obtained from the 
calibration channel fit to data. In order to take corre- 
lations between parameters into account, the full corre- 
lation matrix is used when varying the parameters. All 
PDF parameters that are originally fixed in the fit are 
then varied in turn, and each difference from the nominal 
fit is combined in quadrature and taken as a systematic 
contribution. 

Interference effects between the if* (892) and spin-0 fi- 
nal states (nonresonant and -K)*°(1430)) integrate to zero 
if the acceptance of the detector and analysis is uniform; 
the same is true of the interference between the if* (892) 
and spin-2 final states (iq (1430)). Studies of MC events 
show the efficiency variations are small enough to make 
these interference effects insignificant. The integrated in- 
terference between (892) and other spin-1 amplitudes 
such as if* (1410) is in principle nonzero, but in prac- 
tice is negligible due to the small branching fraction of 
K*°(U10) -> K+ir- (6.6 ± 1.3% [ll|) and the fact that 
the Kit mass lineshapes have little overlap. 

The CP-violating asymmetries for the decays B° — > 
K*°K+K-, B° -> K* q tt+K-, and B° -> K*°tt+tt~ are 
shown in Table U and are consistent with zero. The back- 
ground asymmetries A K f KK , A^ Kv , and A^?^, which 



TABLE II: Summary of systematic uncertainty contributions 
to the branching fraction measurements B° — > K*®h\K^ . 
Multiplicative errors are shown as a percentage of the branch- 
ing fraction and additive errors are shown in events. The final 
row shows the total systematic error on the branching frac- 
tion. 



Error K*°K+K~ K* tt+K~ K*°K+tt- K*°tt+tt- 



source error error error error 

Multiplicative errors (%) 

Tracking 2.0 2.0 2.0 2.0 

Efficiency 3.8 3.9 3.9 4.2 

No. of BB IT IT LI Ll_ 

Tot. mult.(%) 4.4 4.5 4.5 4.7 
Additive errors (events) 
Fit Bias 14 10 2 43 
B-background 17 13 13 26 
Higher K*° bkg 58 118 
Signal mis-id 6 11 
PDF params. 20 26 4 19 



, . C 65 31 18 130 

(events) 

Total (10~ b ) 2.2 0.8 0.5 4.3 



are expected to be consistent with zero, are measured to 
be 0.017 ± 0.010, 0.007 ± 0.004, and 0.0018 ± 0.0033, 
respectively. 

The systematic error on AK*hih 2 is calculated by con- 
sidering contributions due to track finding, fit biases, B- 
background uncertainties and particle interaction asym- 
metries. The error due to fit biases is found to be neg- 
ligible. Tracking efficiency uncertainties are assigned by 
comparing the total number of reconstructed tracks for 
control channels in data and MC simulation. The interac- 
tion asymmetry of matter and antimatter with the detec- 
tor is studied using MC, where biases between -0.01 and 
-0.03 are observed and applied as corrections to the data. 
The uncertainty on the correction, obtained from the cal- 
ibration channel asymmetry difference between MC and 
data, is added as a systematic uncertainty. The contri- 
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bution from B-background is calculated by varying the 
number of expected events within errors and by conserva- 
tively assuming a large CP- violating asymmetry of ± 0.2, 
as there are no available measurements for these decays. 

In summary we analyze K + ir~ h^h^ final states 
to obtain branching fraction and CP-asymmetry mea- 
surements for the decays B° — > K*°K + K~ , B° — > 
K*°ir+K- and B° -> K* 7r+Tr-. We find the results 
to be consistent with the Standard Model, observing the 
decay B° — ► K*°ir + K~ with a 5.3 standard deviation 
significance and placing an upper limit on the branch- 
ing fraction B° — > K*°K + ir~ . We find no evidence for 
CP-violation. 
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